##      dataset          method        trial           mse           
##  airfoil :450   lasso    :400   Min.   : 0.0   Min.   : 0.001183  
##  concrete:450   rand     :400   1st Qu.:12.0   1st Qu.: 0.471746  
##  enc     :450   tourn    :400   Median :24.5   Median : 7.278047  
##  enh     :450   lex      :400   Mean   :24.5   Mean   :      Inf  
##  housing :450   afp      :400   3rd Qu.:37.0   3rd Qu.:21.837452  
##  tower   :450   dc       :400   Max.   :49.0   Max.   :      Inf  
##  uball5d :450   ep-lex-s :400                                     
##  yacht   :450   ep-lex-sd:400                                     
##                 ep-lex-d :400                                     
##        r2                  time          
##  Min.   :-1.323e+31   Min.   :     0.01  
##  1st Qu.: 1.000e+00   1st Qu.: 45412.01  
##  Median : 1.000e+00   Median :103463.23  
##  Mean   :-2.635e+28   Mean   : 79970.78  
##  3rd Qu.: 1.000e+00   3rd Qu.:118045.82  
##  Max.   : 1.000e+00   Max.   :131985.44  
##  NA's   :1                               
##                                          
## 
##      dataset          method        trial           mse           
##  airfoil :450   lasso    :400   Min.   : 0.0   Min.   : 0.001183  
##  concrete:450   rand     :400   1st Qu.:12.0   1st Qu.: 0.471746  
##  enc     :450   tourn    :400   Median :24.5   Median : 7.278047  
##  enh     :450   lex      :400   Mean   :24.5   Mean   :      Inf  
##  housing :450   afp      :400   3rd Qu.:37.0   3rd Qu.:21.837452  
##  tower   :450   dc       :400   Max.   :49.0   Max.   :      Inf  
##  uball5d :450   ep-lex-s :400                                     
##  yacht   :450   ep-lex-sd:400                                     
##                 ep-lex-d :400                                     
##        r2                  time              mse_rank
##  Min.   :-1.323e+31   Min.   :     0.01   Min.   :1  
##  1st Qu.: 1.000e+00   1st Qu.: 45412.01   1st Qu.:3  
##  Median : 1.000e+00   Median :103463.23   Median :5  
##  Mean   :-2.635e+28   Mean   : 79970.78   Mean   :5  
##  3rd Qu.: 1.000e+00   3rd Qu.:118045.82   3rd Qu.:7  
##  Max.   : 1.000e+00   Max.   :131985.44   Max.   :9  
##  NA's   :1                                           
##                                                      
## 
## Warning: Removed 1 rows containing non-finite values (stat_boxplot).
## notch went outside hinges. Try setting notch=FALSE.

## TableGrob (3 x 3) "arrange": 9 grobs
##   z     cells    name           grob
## 1 1 (1-1,1-1) arrange gtable[layout]
## 2 2 (1-1,2-2) arrange gtable[layout]
## 3 3 (1-1,3-3) arrange gtable[layout]
## 4 4 (2-2,1-1) arrange gtable[layout]
## 5 5 (2-2,2-2) arrange gtable[layout]
## 6 6 (2-2,3-3) arrange gtable[layout]
## 7 7 (3-3,1-1) arrange gtable[layout]
## 8 8 (3-3,2-2) arrange gtable[layout]
## 9 9 (3-3,3-3) arrange gtable[layout]

Timing

Stats

## [1] "airfoil"
## 
##  Pairwise comparisons using Wilcoxon rank sum test 
## 
## data:  tmp$mse and tmp$method 
## 
##           lasso   rand    tourn   lex     afp     dc      ep-lex-s
## rand      3.2e-13 -       -       -       -       -       -       
## tourn     0.00013 2.4e-13 -       -       -       -       -       
## lex       0.57427 3.3e-11 8.1e-05 -       -       -       -       
## afp       5.4e-15 3.1e-16 8.1e-05 4.1e-14 -       -       -       
## dc        2.5e-16 2.5e-16 8.1e-11 3.8e-16 0.00051 -       -       
## ep-lex-s  2.5e-16 2.5e-16 2.5e-15 2.7e-16 5.3e-14 8.4e-13 -       
## ep-lex-d  2.5e-16 2.5e-16 2.5e-16 2.5e-16 3.8e-15 1.3e-13 0.17792 
## ep-lex-sd 2.5e-16 2.5e-16 2.5e-16 2.5e-16 2.5e-15 1.6e-14 0.57427 
##           ep-lex-d
## rand      -       
## tourn     -       
## lex       -       
## afp       -       
## dc        -       
## ep-lex-s  -       
## ep-lex-d  -       
## ep-lex-sd 0.57427 
## 
## P value adjustment method: holm 
## [1] "concrete"
## 
##  Pairwise comparisons using Wilcoxon rank sum test 
## 
## data:  tmp$mse and tmp$method 
## 
##           lasso   rand    tourn   lex     afp     dc      ep-lex-s
## rand      2.4e-07 -       -       -       -       -       -       
## tourn     1.5e-14 5.6e-15 -       -       -       -       -       
## lex       9.1e-05 3.1e-10 2.5e-06 -       -       -       -       
## afp       2.5e-16 2.7e-16 7.8e-05 5.4e-14 -       -       -       
## dc        2.5e-16 2.5e-16 1.6e-09 9.5e-16 0.00353 -       -       
## ep-lex-s  2.5e-16 2.5e-16 1.3e-11 5.5e-16 3.9e-06 0.05014 -       
## ep-lex-d  2.5e-16 2.5e-16 9.0e-10 1.5e-15 0.00059 0.34236 0.53724 
## ep-lex-sd 2.5e-16 2.5e-16 6.2e-13 4.2e-16 2.7e-08 0.00017 0.10381 
##           ep-lex-d
## rand      -       
## tourn     -       
## lex       -       
## afp       -       
## dc        -       
## ep-lex-s  -       
## ep-lex-d  -       
## ep-lex-sd 0.05702 
## 
## P value adjustment method: holm 
## [1] "enc"
## 
##  Pairwise comparisons using Wilcoxon rank sum test 
## 
## data:  tmp$mse and tmp$method 
## 
##           lasso   rand    tourn   lex     afp     dc      ep-lex-s
## rand      4.4e-14 -       -       -       -       -       -       
## tourn     6.2e-05 3.3e-14 -       -       -       -       -       
## lex       6.4e-05 3.1e-14 0.99826 -       -       -       -       
## afp       1.5e-12 1.9e-15 0.00462 2.7e-05 -       -       -       
## dc        2.5e-16 2.5e-16 9.4e-14 4.7e-16 3.4e-11 -       -       
## ep-lex-s  2.5e-16 2.5e-16 9.3e-15 6.3e-16 9.8e-13 0.00029 -       
## ep-lex-d  2.5e-16 2.5e-16 6.6e-15 1.0e-15 1.6e-13 4.9e-07 0.99826 
## ep-lex-sd 5.1e-16 2.5e-16 4.1e-14 2.6e-15 1.7e-12 0.00161 0.99826 
##           ep-lex-d
## rand      -       
## tourn     -       
## lex       -       
## afp       -       
## dc        -       
## ep-lex-s  -       
## ep-lex-d  -       
## ep-lex-sd 0.49349 
## 
## P value adjustment method: holm 
## [1] "enh"
## 
##  Pairwise comparisons using Wilcoxon rank sum test 
## 
## data:  tmp$mse and tmp$method 
## 
##           lasso   rand    tourn   lex     afp     dc      ep-lex-s
## rand      4.5e-16 -       -       -       -       -       -       
## tourn     0.00787 1.1e-11 -       -       -       -       -       
## lex       0.00050 1.8e-15 0.21849 -       -       -       -       
## afp       5.4e-08 2.5e-13 0.02175 0.00021 -       -       -       
## dc        2.5e-16 2.5e-16 7.2e-14 2.5e-16 1.6e-09 -       -       
## ep-lex-s  2.5e-16 2.5e-16 9.0e-15 2.7e-16 3.9e-12 0.00043 -       
## ep-lex-d  3.0e-15 1.6e-15 1.0e-14 3.0e-15 2.7e-13 1.8e-08 0.08608 
## ep-lex-sd 2.5e-16 2.5e-16 2.7e-16 2.5e-16 1.4e-15 2.0e-14 0.00012 
##           ep-lex-d
## rand      -       
## tourn     -       
## lex       -       
## afp       -       
## dc        -       
## ep-lex-s  -       
## ep-lex-d  -       
## ep-lex-sd 0.01283 
## 
## P value adjustment method: holm 
## [1] "housing"
## 
##  Pairwise comparisons using Wilcoxon rank sum test 
## 
## data:  tmp$mse and tmp$method 
## 
##           lasso   rand    tourn   lex     afp     dc      ep-lex-s
## rand      1.0e-09 -       -       -       -       -       -       
## tourn     1.00000 1.6e-06 -       -       -       -       -       
## lex       0.32872 2.7e-06 0.92393 -       -       -       -       
## afp       0.15571 1.9e-09 0.92393 0.00244 -       -       -       
## dc        0.05376 2.3e-10 0.29856 0.00081 1.00000 -       -       
## ep-lex-s  1.0e-05 1.6e-13 0.00198 6.7e-08 0.09768 0.78314 -       
## ep-lex-d  2.1e-06 3.7e-13 0.00081 2.6e-08 0.01799 0.35056 1.00000 
## ep-lex-sd 1.5e-05 6.2e-13 0.00081 3.4e-07 0.01572 0.22273 1.00000 
##           ep-lex-d
## rand      -       
## tourn     -       
## lex       -       
## afp       -       
## dc        -       
## ep-lex-s  -       
## ep-lex-d  -       
## ep-lex-sd 1.00000 
## 
## P value adjustment method: holm 
## [1] "tower"
## 
##  Pairwise comparisons using Wilcoxon rank sum test 
## 
## data:  tmp$mse and tmp$method 
## 
##           lasso   rand    tourn   lex     afp     dc      ep-lex-s
## rand      2.5e-16 -       -       -       -       -       -       
## tourn     6.9e-16 2.0e-14 -       -       -       -       -       
## lex       6.8e-16 4.2e-16 0.00017 -       -       -       -       
## afp       1.4e-15 7.1e-16 0.00638 0.16381 -       -       -       
## dc        1.7e-15 2.5e-16 1.2e-07 0.00085 1.3e-05 -       -       
## ep-lex-s  0.53706 2.5e-16 1.0e-14 7.9e-14 7.9e-14 4.2e-12 -       
## ep-lex-d  0.53706 2.5e-16 6.2e-16 7.7e-16 1.4e-15 8.3e-16 0.34560 
## ep-lex-sd 0.00638 2.5e-16 1.6e-15 6.4e-15 7.6e-15 3.7e-14 0.00638 
##           ep-lex-d
## rand      -       
## tourn     -       
## lex       -       
## afp       -       
## dc        -       
## ep-lex-s  -       
## ep-lex-d  -       
## ep-lex-sd 0.06600 
## 
## P value adjustment method: holm 
## [1] "uball5d"
## 
##  Pairwise comparisons using Wilcoxon rank sum test 
## 
## data:  tmp$mse and tmp$method 
## 
##           lasso   rand    tourn   lex     afp     dc      ep-lex-s
## rand      1.9e-15 -       -       -       -       -       -       
## tourn     3.8e-16 1.0e-11 -       -       -       -       -       
## lex       2.5e-16 7.1e-16 0.878   -       -       -       -       
## afp       2.5e-16 1.9e-14 1.000   0.096   -       -       -       
## dc        2.5e-16 2.5e-16 3.9e-09 3.9e-09 1.4e-13 -       -       
## ep-lex-s  2.5e-16 2.5e-16 2.6e-14 4.0e-13 1.9e-15 1.0e-10 -       
## ep-lex-d  2.5e-16 2.5e-16 1.1e-14 2.3e-13 7.1e-16 1.0e-10 1.000   
## ep-lex-sd 2.5e-16 2.5e-16 4.8e-15 1.0e-13 7.0e-16 1.5e-11 1.000   
##           ep-lex-d
## rand      -       
## tourn     -       
## lex       -       
## afp       -       
## dc        -       
## ep-lex-s  -       
## ep-lex-d  -       
## ep-lex-sd 1.000   
## 
## P value adjustment method: holm 
## [1] "yacht"
## 
##  Pairwise comparisons using Wilcoxon rank sum test 
## 
## data:  tmp$mse and tmp$method 
## 
##           lasso   rand    tourn   lex     afp     dc    ep-lex-s ep-lex-d
## rand      2.4e-12 -       -       -       -       -     -        -       
## tourn     2.5e-16 1.4e-13 -       -       -       -     -        -       
## lex       1.7e-15 1.2e-11 1.000   -       -       -     -        -       
## afp       3.5e-15 1.7e-13 1.000   1.000   -       -     -        -       
## dc        2.5e-16 3.0e-16 5.6e-10 1.2e-08 1.2e-08 -     -        -       
## ep-lex-s  2.5e-16 4.1e-16 2.2e-08 8.8e-08 2.4e-07 1.000 -        -       
## ep-lex-d  2.5e-16 5.3e-16 3.2e-10 8.1e-10 6.2e-10 0.104 0.230    -       
## ep-lex-sd 2.5e-16 5.5e-16 1.5e-07 7.9e-07 4.9e-06 1.000 1.000    0.053   
## 
## P value adjustment method: holm
## lasso & rand & tourn & lex & afp & dc & ep-lex-s & ep-lex-d\\ 
## airfoil & 2.54e-16 & 2.54e-16 & 2.54e-16 & 2.54e-16 & 2.55e-15 & 1.59e-14 & 0.574 & 0.574\\ 
## concrete & 2.54e-16 & 2.54e-16 & 6.24e-13 & 4.25e-16 & 2.74e-08 & 0.000166 & 0.104 & 0.057\\ 
## enc & 5.15e-16 & 2.54e-16 & 4.12e-14 & 2.57e-15 & 1.67e-12 & 0.00161 & 0.998 & 0.493\\ 
## enh & 2.54e-16 & 2.54e-16 & 2.67e-16 & 2.54e-16 & 1.41e-15 & 2e-14 & 0.000121 & 0.0128\\ 
## housing & 1.51e-05 & 6.2e-13 & 0.000812 & 3.4e-07 & 0.0157 & 0.223 & 1 & 1\\ 
## tower & 0.00638 & 2.54e-16 & 1.57e-15 & 6.39e-15 & 7.63e-15 & 3.67e-14 & 0.00638 & 0.066\\ 
## uball5d & 2.54e-16 & 2.54e-16 & 4.8e-15 & 1.04e-13 & 6.96e-16 & 1.55e-11 & 1 & 1\\ 
## yacht & 2.54e-16 & 5.46e-16 & 1.52e-07 & 7.86e-07 & 4.93e-06 & 1 & 1 & 0.053\\
## % latex table generated in R 3.4.1 by xtable 1.8-2 package
## % Tue Dec 12 11:29:53 2017
## \begin{table}[ht]
## \centering
## \caption{Significance test $p$-values comparing test MSE using the pair-wise Wilcoxon rank-sum test with Holm correction for multiple comparisons. All significance tests are conducted relative to semi-dynamic $\epsilon$-lexicase (ep-lex-sd). Bold indicates $p<$ 0.05.} 
## \label{tbl:wilcox}
## \begingroup\footnotesize
## \begin{tabular}{rllllllll}
##   \toprule
##  & lasso & rand & tourn & lex & afp & dc & ep-lex-s & ep-lex-d \\ 
##   \midrule
## airfoil & {\bf 2.54e-16} & {\bf 2.54e-16} & {\bf 2.54e-16} & {\bf 2.54e-16} & {\bf 2.55e-15} & {\bf 1.59e-14} & 0.57 & 0.57 \\ 
##   concrete & {\bf 2.54e-16} & {\bf 2.54e-16} & {\bf 6.24e-13} & {\bf 4.25e-16} & {\bf 2.74e-08} & {\bf 1.66e-04} & 0.1 & 0.057 \\ 
##   enc & {\bf 5.15e-16} & {\bf 2.54e-16} & {\bf 4.12e-14} & {\bf 2.57e-15} & {\bf 1.67e-12} & {\bf 1.61e-03} &   1 & 0.49 \\ 
##   enh & {\bf 2.54e-16} & {\bf 2.54e-16} & {\bf 2.67e-16} & {\bf 2.54e-16} & {\bf 1.41e-15} & {\bf 2.00e-14} & {\bf 1.21e-04} & {\bf 1.28e-02} \\ 
##   housing & {\bf 1.51e-05} & {\bf 6.20e-13} & {\bf 8.12e-04} & {\bf 3.40e-07} & {\bf 1.57e-02} & 0.22 &   1 &   1 \\ 
##   tower & {\bf 6.38e-03} & {\bf 2.54e-16} & {\bf 1.57e-15} & {\bf 6.39e-15} & {\bf 7.63e-15} & {\bf 3.67e-14} & {\bf 6.38e-03} & 0.066 \\ 
##   uball5d & {\bf 2.54e-16} & {\bf 2.54e-16} & {\bf 4.80e-15} & {\bf 1.04e-13} & {\bf 6.96e-16} & {\bf 1.55e-11} &   1 &   1 \\ 
##   yacht & {\bf 2.54e-16} & {\bf 5.46e-16} & {\bf 1.52e-07} & {\bf 7.86e-07} & {\bf 4.93e-06} &   1 &   1 & 0.053 \\ 
##    \bottomrule
## \end{tabular}
## \endgroup
## \end{table}

##      dataset        method        N         mse_rank           sd        
##  airfoil :9   lasso    :8   Min.   :50   Min.   :1.640   Min.   :0.1414  
##  concrete:9   rand     :8   1st Qu.:50   1st Qu.:2.910   1st Qu.:0.8114  
##  enc     :9   tourn    :8   Median :50   Median :5.480   Median :1.1606  
##  enh     :9   lex      :8   Mean   :50   Mean   :5.000   Mean   :1.1455  
##  housing :9   afp      :8   3rd Qu.:50   3rd Qu.:6.625   3rd Qu.:1.3690  
##  tower   :9   dc       :8   Max.   :50   Max.   :8.980   Max.   :2.3592  
##  uball5d :9   ep-lex-s :8                                                
##  yacht   :9   ep-lex-sd:8                                                
##               ep-lex-d :8                                                
##        se               ci         
##  Min.   :0.0200   Min.   :0.04019  
##  1st Qu.:0.1147   1st Qu.:0.23060  
##  Median :0.1641   Median :0.32983  
##  Mean   :0.1620   Mean   :0.32554  
##  3rd Qu.:0.1936   3rd Qu.:0.38907  
##  Max.   :0.3336   Max.   :0.67047  
##                                    
##                                    
## 
## 
##  Friedman rank sum test
## 
## data:  mse_rank and method and dataset
## Friedman chi-squared = 54.8, df = 8, p-value = 4.828e-09
## Loading required package: coin
## Loading required package: survival
## Loading required package: multcomp
## Loading required package: mvtnorm
## Loading required package: TH.data
## Loading required package: MASS
## 
## Attaching package: 'TH.data'
## The following object is masked from 'package:MASS':
## 
##     geyser
## Loading required package: colorspace

## $Friedman.Test
## 
##  Asymptotic General Symmetry Test
## 
## data:  mse_rank by
##   method (lasso, rand, tourn, lex, afp, dc, ep-lex-s, ep-lex-sd, ep-lex-d) 
##   stratified by dataset
## maxT = 5.1121, p-value = 8.702e-06
## alternative hypothesis: two.sided
## 
## 
## $PostHoc.Test
##                                  
## rand - lasso         9.856379e-01
## tourn - lasso        9.983731e-01
## lex - lasso          9.999510e-01
## afp - lasso          9.096883e-01
## dc - lasso           2.061212e-01
## ep-lex-s - lasso     1.559229e-02
## ep-lex-sd - lasso    1.379760e-03
## ep-lex-d - lasso     2.992708e-03
## tourn - rand         7.252481e-01
## lex - rand           8.737233e-01
## afp - rand           2.981041e-01
## dc - rand            1.105767e-02
## ep-lex-s - rand      2.312908e-04
## ep-lex-sd - rand     6.674121e-06
## ep-lex-d - rand      3.721667e-05
## lex - tourn          9.999991e-01
## afp - tourn          9.993834e-01
## dc - tourn           6.648655e-01
## ep-lex-s - tourn     1.339409e-01
## ep-lex-sd - tourn    2.091101e-02
## ep-lex-d - tourn     3.730903e-02
## afp - lex            9.923325e-01
## dc - lex             4.732915e-01
## ep-lex-s - lex       6.453383e-02
## ep-lex-sd - lex      7.675381e-03
## ep-lex-d - lex       1.559229e-02
## dc - afp             9.594898e-01
## ep-lex-s - afp       4.732915e-01
## ep-lex-sd - afp      1.339409e-01
## ep-lex-d - afp       2.061212e-01
## ep-lex-s - dc        9.923325e-01
## ep-lex-sd - dc       8.306512e-01
## ep-lex-d - dc        9.096883e-01
## ep-lex-sd - ep-lex-s 9.993834e-01
## ep-lex-d - ep-lex-s  9.999510e-01
## ep-lex-d - ep-lex-sd 1.000000e+00
## $Friedman.Test
## 
##  Asymptotic General Symmetry Test
## 
## data:  mse_rank by
##   method (lasso, rand, tourn, lex, afp, dc, ep-lex-s, ep-lex-sd, ep-lex-d) 
##   stratified by dataset
## maxT = 5.1121, p-value = 1.322e-05
## alternative hypothesis: two.sided
## 
## 
## $PostHoc.Test
##                                  
## rand - lasso         9.856379e-01
## tourn - lasso        9.983731e-01
## lex - lasso          9.999510e-01
## afp - lasso          9.096883e-01
## dc - lasso           2.061212e-01
## ep-lex-s - lasso     1.559229e-02
## ep-lex-sd - lasso    1.379760e-03
## ep-lex-d - lasso     2.992708e-03
## tourn - rand         7.252481e-01
## lex - rand           8.737233e-01
## afp - rand           2.981041e-01
## dc - rand            1.105767e-02
## ep-lex-s - rand      2.312908e-04
## ep-lex-sd - rand     6.674121e-06
## ep-lex-d - rand      3.721667e-05
## lex - tourn          9.999991e-01
## afp - tourn          9.993834e-01
## dc - tourn           6.648655e-01
## ep-lex-s - tourn     1.339409e-01
## ep-lex-sd - tourn    2.091101e-02
## ep-lex-d - tourn     3.730903e-02
## afp - lex            9.923325e-01
## dc - lex             4.732915e-01
## ep-lex-s - lex       6.453383e-02
## ep-lex-sd - lex      7.675381e-03
## ep-lex-d - lex       1.559229e-02
## dc - afp             9.594898e-01
## ep-lex-s - afp       4.732915e-01
## ep-lex-sd - afp      1.339409e-01
## ep-lex-d - afp       2.061212e-01
## ep-lex-s - dc        9.923325e-01
## ep-lex-sd - dc       8.306512e-01
## ep-lex-d - dc        9.096883e-01
## ep-lex-sd - ep-lex-s 9.993834e-01
## ep-lex-d - ep-lex-s  9.999510e-01
## ep-lex-d - ep-lex-sd 1.000000e+00
## 
## Call:
## lm(formula = mse_rank ~ method, data = rank_stats)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.1925 -0.3806  0.0387  0.3587  1.8875 
## 
## Coefficients:
##                 Estimate Std. Error t value Pr(>|t|)    
## (Intercept)       7.0725     0.2991  23.650  < 2e-16 ***
## methodrand        1.4950     0.4229   3.535 0.000771 ***
## methodtourn      -0.9925     0.4229  -2.347 0.022095 *  
## methodlex        -0.6375     0.4229  -1.507 0.136716    
## methodafp        -1.6425     0.4229  -3.884 0.000249 ***
## methoddc         -3.0575     0.4229  -7.229 7.95e-10 ***
## methodep-lex-s   -4.3700     0.4229 -10.333 3.44e-15 ***
## methodep-lex-sd  -4.7925     0.4229 -11.332  < 2e-16 ***
## methodep-lex-d   -4.6550     0.4229 -11.007 2.60e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.8458 on 63 degrees of freedom
## Multiple R-squared:  0.8785, Adjusted R-squared:  0.8631 
## F-statistic: 56.96 on 8 and 63 DF,  p-value: < 2.2e-16
## Analysis of Variance Table
## 
## Response: mse_rank
##           Df Sum Sq Mean Sq F value    Pr(>F)    
## method     8 325.99  40.749  56.956 < 2.2e-16 ***
## Residuals 63  45.07   0.715                      
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## Call:
##    aov(formula = mse_rank ~ method, data = rank_stats)
## 
## Terms:
##                   method Residuals
## Sum of Squares  325.9942   45.0738
## Deg. of Freedom        8        63
## 
## Residual standard error: 0.845847
## Estimated effects may be unbalanced
## [1] "ep-lex-s-lasso"
## [1] "{\\bf 1.55e-11}"
## [1] "ep-lex-s-rand"
## [1] "{\\bf 1.53e-11}"
## [1] "ep-lex-s-tourn"
## [1] "{\\bf 1.36e-09}"
## [1] "ep-lex-s-lex"
## [1] "{\\bf 6.19e-11}"
## [1] "ep-lex-s-afp"
## [1] "{\\bf 6.32e-07}"
## [1] "ep-lex-s-dc"
## [1] "0.066"
## [1] "ep-lex-s-ep-lex-s"
## [1] NA
## [1] "ep-lex-s-ep-lex-sd"
## [1] NA
## [1] "ep-lex-sd-lasso"
## [1] "{\\bf 1.54e-11}"
## [1] "ep-lex-sd-rand"
## [1] "{\\bf 1.53e-11}"
## [1] "ep-lex-sd-tourn"
## [1] "{\\bf 4.00e-11}"
## [1] "ep-lex-sd-lex"
## [1] "{\\bf 1.63e-11}"
## [1] "ep-lex-sd-afp"
## [1] "{\\bf 1.17e-08}"
## [1] "ep-lex-sd-dc"
## [1] "{\\bf 3.59e-03}"
## [1] "ep-lex-sd-ep-lex-s"
## [1] "0.98"
## [1] "ep-lex-sd-ep-lex-sd"
## [1] NA
## [1] "ep-lex-d-lasso"
## [1] "{\\bf 1.54e-11}"
## [1] "ep-lex-d-rand"
## [1] "{\\bf 1.53e-11}"
## [1] "ep-lex-d-tourn"
## [1] "{\\bf 1.05e-10}"
## [1] "ep-lex-d-lex"
## [1] "{\\bf 1.86e-11}"
## [1] "ep-lex-d-afp"
## [1] "{\\bf 4.32e-08}"
## [1] "ep-lex-d-dc"
## [1] "{\\bf 1.00e-02}"
## [1] "ep-lex-d-ep-lex-s"
## [1] "  1"
## [1] "ep-lex-d-ep-lex-sd"
## [1] "  1"
## % latex table generated in R 3.4.1 by xtable 1.8-2 package
## % Tue Dec 12 11:29:57 2017
## \begin{table}[ht]
## \centering
## \caption{Post-hoc pairwise statistical tests of rankings across problems according to Tukey's Honest Significant Difference test. Bold values indicate $p<$ 0.05 with adjustment for multiple comparisons.} 
## \label{tbl:hsd}
## \begin{tabular}{rllllllll}
##   \toprule
##  & lasso & rand & tourn & lex & afp & dc & ep-lex-s & ep-lex-sd \\ 
##   \midrule
## ep-lex-s & {\bf 1.55e-11} & {\bf 1.53e-11} & {\bf 1.36e-09} & {\bf 6.19e-11} & {\bf 6.32e-07} & 0.066 &  &  \\ 
##   ep-lex-sd & {\bf 1.54e-11} & {\bf 1.53e-11} & {\bf 4.00e-11} & {\bf 1.63e-11} & {\bf 1.17e-08} & {\bf 3.59e-03} & 0.98 &  \\ 
##   ep-lex-d & {\bf 1.54e-11} & {\bf 1.53e-11} & {\bf 1.05e-10} & {\bf 1.86e-11} & {\bf 4.32e-08} & {\bf 1.00e-02} &   1 &   1 \\ 
##    \bottomrule
## \end{tabular}
## \end{table}

Convergence

##        method         dataset         trial            gen      
##  rand     :8400   airfoil :8400   Min.   : 0.00   Min.   :   0  
##  tourn    :8400   concrete:8400   1st Qu.:12.00   1st Qu.: 250  
##  lex      :8400   enc     :8400   Median :25.00   Median : 500  
##  afp      :8400   enh     :8400   Mean   :24.53   Mean   : 500  
##  dc       :8400   housing :8400   3rd Qu.:37.00   3rd Qu.: 750  
##  ep-lex-s :8400   tower   :8400   Max.   :49.00   Max.   :1000  
##  ep-lex-sd:8400   uball5d :8400                                 
##  ep-lex-d :8400   yacht   :8400                                 
##     best_fit            novelty      
##  Min.   :0.0009085   Min.   :0.0010  
##  1st Qu.:0.0302705   1st Qu.:0.5730  
##  Median :0.1385115   Median :0.8060  
##  Mean   :0.1733712   Mean   :0.6973  
##  3rd Qu.:0.2447150   3rd Qu.:0.8980  
##  Max.   :0.9668800   Max.   :1.0000  
##                                      
## 
## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals

## Warning: position_dodge requires non-overlapping x intervals